Augmented reality medical diagnostic projection

ABSTRACT

Methods, computer-readable media, and apparatuses for presenting medical records associated with a body part of person via an augmented reality device are described. For example, a processing system including at least one processor may identify at least one body part of a person in a visual data feed of an augmented reality device, obtain based on the identifying at least a first medical record of the person that is associated with the at least one body part, obtain at least a second medical record of at least one relative of the person that is associated with the at least one body part, and present, via the augmented reality device, at least the first medical record and the least the second medical record.

The present disclosure relates generally to augmented reality systems,and more particularly to methods, computer-readable media, andapparatuses for presenting medical records associated with a body partof a person via an augmented reality device.

SUMMARY

Methods, computer-readable media, and apparatuses for presenting medicalrecords associated with a body part of a person via an augmented realitydevice are described. For example, a processing system including atleast one processor may identify at least one body part of a person in avisual data feed of an augmented reality device, obtain based on theidentifying at least a first medical record of the person that isassociated with the at least one body part, obtain at least a secondmedical record of at least one relative of the person that is associatedwith the at least one body part, and present, via the augmented realitydevice, at least the first medical record and at least the secondmedical record.

BRIEF DESCRIPTION OF THE DRAWINGS

The teachings of the present disclosure can be readily understood byconsidering the following detailed description in conjunction with theaccompanying drawings, in which:

FIG. 1 illustrates an example network related to the present disclosure;

FIG. 2 illustrates an example process in accordance with the presentdisclosure;

FIG. 3 illustrates a flowchart of an example method for presentingmedical records associated with a body part of a person via an augmentedreality device; and

FIG. 4 illustrates a high level block diagram of a computing devicespecifically programmed to perform the steps, functions, blocks and/oroperations described herein.

To facilitate understanding, identical reference numerals have beenused, where possible, to designate identical elements that are common tothe figures.

DETAILED DESCRIPTION

Examples of the present disclosure include superimposing a user's (e.g.,a patient's) own medical history and family medical history over imagingof an augmented reality device, in addition to information regardingsimilar cases of other patients with the same circumstances to helpdiagnose and treat various medical conditions. In one example, thepresent disclosure provides an interactive model via augmented reality(AR) between a patient and a healthcare provider. In one example,systems of the present disclosure may provide diagnostic predictions,and may also search for and recommend medical providers based on thediagnostic predictions.

In one example, a patient's historical medical records are digitized andform layers that are superimposed on the current imaging of an augmentedreality (AR) device. Each event (e.g., a bone fracture, an infection, aprior surgical procedure performed on a body part of interest, a cancertissue, etc.) may comprise a layer, where layers are historical and/orrelated to events. In one example, a last layer may include a diagnosticprediction that is determined via a machine learning module thatcompares other patients' results and progressions of one or more medicalconditions. For instance, in one example, a system of the presentdisclosure may correlate a present visual feed (e.g., visual datarelating to a certain part of the body captured in images and/or videofrom an AR device) with an event. To illustrate, a doctor may be lookingat a patient's arm with an AR device (e.g., an AR headset/glasses). Thesystem may then superimpose layers into the doctor's field of view viathe AR device containing medical records related to the arm. This mayinclude relevant information related to the arm (e.g., medical historiesof the patient and of the patient's relatives (broadly any person who isconnected with another or others by blood or marriage, e.g., parents,siblings, uncles, aunts, grandparents, cousins, nieces, nephews and soon), images, potential diagnoses, drugs, treatments, and so forth). Inone example, the AR device may recognize various body parts of a human,and may perform the correlation and obtain other matching informationfrom the patient's history, the history of the patient's family, as wellas from epidemiologic data (e.g., anonymized statistical data fromvarious patients in a geographic area, or other demographic grouping),anonymized case studies, or indirectly related events (e.g., knowledgeof possible exposure to a certain bacteria due to a recent visit to aparticular healthcare facility, knowledge of recent travel to aparticular region, etc.). In another example, the AR device may forwarda visual feed to a network-based processing system to perform the sameor similar functions, and the results may be provided by the processingsystem to the AR device for presentation.

In either case, systems of the present disclosure may identifyorgans/body parts via the AR device through the following mechanisms:shapes of body parts for different ages, genders, etc., and also withanomalies in shapes as a result of disease, birth defect, and/orgenetics, the location of each organ in the body, relative position(s)to other body parts, and the connectivity to other organs, the flow offluids in/out of an organ, e.g., heart, lungs, liver, kidney, etc., theperiodic movements of an organ (e.g., heart beats,inhalation/exhalation, etc.) In one example, a healthcare provider or aself-examining person may also bias the system to identify a part of thebody.

In one example, the AR device may respond to voice commands or otheruser inputs to zoom in with respect to certain areas of the body and tobring/remove layers from the overlay on the field of view. In oneexample, systems of the present disclosure may also be used bynon-medical professionals for self-examination and self-diagnosis. Forinstance, a user may view a portion of his/her body via an AR device.Then, the AR device and/or other network-based components of the systemmay provide medical history information and diagnostic predictions, andmay also recommend to connect with certain medical professionals, e.g.,specialists associated with one or more of the diagnostic predictions.In one example, the AR device and/or other system component(s) mayautomatically connect with a device of medical professional to providethe medical professional with the same visual feed as well as theadditional information, e.g., user/patient medical history, familymedical history, diagnostic prediction(s), etc. In one example, themedical professional may be granted remote control via his or her deviceto select/deselect layers, to change the focus of the area on the body,to bias the system for certain suspected conditions, and so forth. Thisallows the medical professional to remotely view the body part ofinterest overlaid with the relevant medical information.

In one example, the system may also provide recommended drug treatments,recommended therapy procedures, recommended surgical procedures, orother recommended interventions based upon one or more diagnosticpredictions. For instance, the system may store or have access to one ormore databases which store correlations between diagnostic predictionsand possible courses of treatment. In one example, systems of thepresent disclosure may include one or more machine learning (ML) modelswhich may use pattern recognition to find similar cases to the conditionor diagnostic prediction from one or more data repositories. Forexample, the ML models may use any classification and patternrecognition set of algorithms, such as classification supervisedalgorithms and clustering unsupervised algorithms, to predictcategorical labels, and multilinear subspace learning algorithms topredict labels of multi-dimensional data.

To illustrate, information from a visual feed from the AR device,user/patient history, family history, and possible user input from theuser/patient and/or medical professional may comprise inputs to the MLmodel(s), which may output potential diagnosis and treatment plans. Inone example, the system may use a profile for a user/patient withcurrent conditions, past conditions, family history, and/or medicalprofessional input. The system may then search through one or more datarepositories for similar profiles with the highest matching scores. Forinstance, more similar situations correspond to higher matching scoresand vice versa. In addition, similar cases may have more weight in thedecision making process for determining a diagnostic predictions. Forexample, if there is a 40 year old male subject and the data repositorycontains a first event record for a 38 year old male subject with allthe conditions matched and a second event record for 28 year old malesample patient with all the conditions matched, the diagnosis from theevent related to the 38 year old may have more impact or relevance inthe decision by the ML model than the 28 years old's case. Aside fromage, other parameters may indicate additional relevance between thesubjects, e.g., similar family medical history, similar ethicbackground, similar diet (e.g., smokers versus non-smokers), similarrelated environmental working conditions (e.g., working in a coal mine,working in a factory, working in an office building, working on a ship,working on night shifts, etc.) and so on. These and other aspects of thepresent disclosure are described in greater detail below in connectionwith the examples of FIGS. 1-4.

To further aid in understanding the present disclosure, FIG. 1illustrates an example system 100 in which examples of the presentdisclosure for presenting medical records associated with a body part ofa person via an augmented reality device may operate. The system 100 mayinclude any one or more types of communication networks, such as atraditional circuit switched network (e.g., a public switched telephonenetwork (PSTN)) or a packet network such as an Internet Protocol (IP)network (e.g., an IP Multimedia Subsystem (IMS) network), anasynchronous transfer mode (ATM) network, a wireless network, a cellularnetwork (e.g., 2G, 3G, 4G, 5G and the like), a long term evolution (LTE)network, and the like, related to the current disclosure. It should benoted that an IP network is broadly defined as a network that usesInternet Protocol to exchange data packets. Additional example IPnetworks include Voice over IP (VoIP) networks, Service over IP (SoIP)networks, and the like.

In one example, the system 100 may comprise a telecommunication network102. The telecommunication network 102 may be in communication with oneor more access networks 120 and 122, and the Internet (not shown). Inone example, telecommunication network 102 may combine core networkcomponents of a cellular network with components of a triple playservice network; where triple-play services include telephone services,Internet services and television services to subscribers. For example,telecommunication network 102 may functionally comprise a fixed mobileconvergence (FMC) network, e.g., an IP Multimedia Subsystem (IMS)network. In addition, telecommunication network 102 may functionallycomprise a telephony network, e.g., an Internet Protocol/Multi-ProtocolLabel Switching (IP/MPLS) backbone network utilizing Session InitiationProtocol (SIP) for circuit-switched and Voice over Internet Protocol(VoIP) telephony services. Telecommunication network 102 may furthercomprise a broadcast television network, e.g., a traditional cableprovider network or an Internet Protocol Television (IPTV) network, aswell as an Internet Service Provider (ISP) network. In one example,telecommunication network 102 may include a plurality of television (TV)servers (e.g., a broadcast server, a cable head-end), a plurality ofcontent servers, an advertising server (AS), an interactive TV/ video ondemand (VoD) server, and so forth. For ease of illustration, variousadditional elements of network 102 are omitted from FIG. 1.

In one example, the access networks 120 and 122 may comprise DigitalSubscriber Line (DSL) networks, public switched telephone network (PSTN)access networks, broadband cable access networks, Local Area Networks(LANs), wireless access networks (e.g., an Institute for Electrical andElectronics Engineers (IEEE) 802.11/Wi-Fi network and the like),cellular access networks, 3^(rd) party networks, and the like. Forexample, the operator of telecommunication network 102 may provide acable television service, an IPTV service, or any other types oftelecommunication service to subscribers via access networks 120 and122. In one example, the access networks 120 and 122 may comprisedifferent types of access networks, may comprise the same type of accessnetwork, or some access networks may be the same type of access networkand other may be different types of access networks. In one embodiment,the telecommunication network 102 may be operated by a telecommunicationnetwork service provider. The telecommunication network 102 and theaccess networks 120 and 122 may be operated by different serviceproviders, the same service provider or a combination thereof, or may beoperated by entities having core businesses that are not related totelecommunications services, e.g., corporate, governmental oreducational institution LANs, and the like.

In one example, the access networks 120 may be in communication with oneor more devices 110-112. Similarly, access networks 122 may be incommunication with one or more devices, e.g., device 113, server 116,database (DB 118), and so forth. Access networks 120 and 122 maytransmit and receive communications between devices 110-113, betweendevices 110-113, and server 116 and/or database (DB) 118, applicationserver 104 and/or database (DB) 106, other components oftelecommunication network 102, devices reachable via the internet ingeneral, and so forth. In one example, each of the devices 110-113 maycomprise any single device or combination of devices that may comprise auser endpoint device. For example, the devices 110-113 may each comprisea mobile device, a cellular smart phone, a laptop, a tablet computer, adesktop computer, an application server, a bank or cluster of suchdevices, and the like. In one example, devices 110-112 may comprise ARdevices such as heads-up displays, wearable or non-wearable opticalsee-through or video see-through devices, handheld computing deviceswith at least a camera and a display, and so forth. For instance, asillustrated in FIG. 1, devices 110 and 111 may comprise wearablecomputing devices (e.g., smart glasses, augmented reality glasses,headsets, or the like). Similarly, device 112 may comprise a tabletcomputer, cellular smartphone and/or non-cellular wireless device, orthe like with at least a camera and a display.

In one example, devices 110-113 may each comprise programs, logic orinstructions for performing functions in connection with examples of thepresent disclosure for presenting medical records associated with a bodypart of a person via an augmented reality device. For example, devices110-113 may each comprise a computing system or device, such ascomputing system 400 depicted in FIG. 4, and may be configured toprovide one or more operations or functions in connection with examplesof the present disclosure for presenting medical records associated witha body part of a person via an augmented reality device, as describedherein.

In one example, the access networks 122 may also be in communicationwith a server 116 and a database (DB) 118. The server 116 and DB 118 maybe associated with a service, or system for presenting medical recordsassociated with a body part of a person via an augmented reality device,as described herein. In accordance with the present disclosure, server116 may comprise a computing system or server, such as computing system400 depicted in FIG. 4, and may be configured to provide one or moreoperations or functions for presenting medical records associated with abody part of a person via an augmented reality device, as describedherein. It should be noted that as used herein, the terms “configure,”and “reconfigure” may refer to programming or loading a processingsystem with computer-readable/computer-executable instructions, code,and/or programs, e.g., in a distributed or non-distributed memory, whichwhen executed by a processor, or processors, of the processing systemwithin a same device or within distributed devices, may cause theprocessing system to perform various functions. Such terms may alsoencompass providing variables, data values, tables, objects, or otherdata structures or the like which may cause a processing systemexecuting computer-readable instructions, code, and/or programs tofunction differently depending upon the values of the variables or otherdata structures that are provided. As referred to herein a “processingsystem” may comprise a computing device including one or moreprocessors, or cores (e.g., as illustrated in FIG. 4 and discussedbelow) or multiple computing devices collectively configured to performvarious steps, functions, and/or operations in accordance with thepresent disclosure.

In one example, DB 118 may comprise a physical storage device integratedwith server 116 (e.g., a database server), or attached or coupled to theserver 116, to store various types of information in support of systemsfor presenting medical records associated with a body part of a personvia an augmented reality device, in accordance with the presentdisclosure. For example, DB 118 may store individual medical records,may store indications of relations among individuals (e.g., to obtainmedical records of relatives of a person), may store informationregarding patterns for detecting body parts, for detecting particularmotions associated with body parts, for detecting conditions of bodyparts, etc., may store machine learning-based modules (e.g., ML models)for making diagnostic predictions based upon body part conditions ofsubject persons, medical records of the persons, medical records ofrelatives, and/or user inputs, may store information for automaticallycontacting medical providers or other caregivers on behalf of a person,and so forth that may be processed by server 116 or provided to devicesrequesting medical records and/or any other information outlined abovefrom server 116.

To illustrate, server 116 may identify at least one body part of aperson in a visual data feed of an AR device, obtain at least a firstmedical record of the person that is associated with the at least onebody part, obtain at least a second medical record of at least onerelative of the person that is associated with the at least one bodypart, and present, via the augmented reality device, at least the firstmedical record and at least the second medical record. For example, thepresenting may include projecting a transparent overlay of at least thefirst medical record and at least the second medical record via the ARdevice.

In one example, the AR device may comprise one of the devices 110-112,and the body part may be of one of the persons 190, 199, or 192,respectively. In one example, a user, e.g., person 190 or 192 may beexamining himself or herself directly with his or her own device 110 or112. For instance, the respective fields of view 180 and 182 may includethe hands of users 190 and 192, respectively. In another example, the ARdevice may be used by a doctor or other caregivers, or simply a secondperson viewing a first person with the AR device, e.g., person 191viewing person 199 via device 111. For instance, the field of view 181via device 111 may include a hand of the person 199. In this regard, thepresenting may include providing the medical records and instructions byserver 116 to the AR device to cause the medical records to be projectedby the AR device via a transparent overlay, e.g., in one of field ofviews 180-182. For instance, devices 110 and 111 may present the medicalinformation via projector(s) and reflector(s) or the like, while device112 may present the medical information via a display screen that isalso presenting the information from the visual feed, e.g., the hand andarm of user 192.

In one example, the server 116 may identify the at least one body partvia a machine learning-based pattern detection in accordance withinformation from a visual data feed from the augmented reality device(e.g., one of the devices 110-112). In one example, the server 116 maydetect a motion associated with the at least one body part, e.g., acough, a heartbeat pattern, a vascular/blood flow pattern, a movementpattern of a mouth during speech, a tremor in the at least one body part(e.g., one or more body parts), and so forth. In one example, the server116 may determine at least one condition associated with the at leastone body part based upon the visual data feed, such as detecting a moleon the skin, a mole pattern, a sunburn or sunburn pattern, a skininfection, a lesion, a swollen joint, swollen skin, etc., or detectingarrhythmia, detecting facial paralysis and/or facial droop, and so on.

In one example, the server 116 may select at least the first medicalrecord from among a plurality of medical records of the person (e.g.,person 190, 192, or 199) and the second medical record from among aplurality of medical records of relatives of the subject person basedupon the at least one body part, such as a hand, and/or a condition ofthe at least one body part, in addition to a user input. For example,the user input may include a suspected condition of the person, apreference for a type of medical record, or a preference for medicalrecords associated with the at least one body part. The user input maycome from the subject person (e.g., person 190, 192, and/or 199 viadevices 111-112, respectively), or may come from another user (e.g., amedical professional, caregiver, or the like, such as person 191 viadevice 111 or another person via device 113). To illustrate, a doctor,e.g., person 191, may suspect that a patient, e.g., person 199, may havea fractured wrist and may provide a user input to focus upon the wrist,medical records associated with the wrist, hands, forearm, or relatedbody parts, and so forth. The user input may comprise verbal commands ornatural language speech which may be captured via device 111 andforwarded to server 116, for example. However, in other, further, anddifferent examples, the user input may be of a different form, such astext input via a keyboard, selection from among a plurality of optionsvia a mouse in connection with a graphical user interface, hand gesturescaptured via a camera of the device 111, and so forth. Accordingly,server 116 may focus upon identifying the at least one body part inaccordance with the user input. For instance, there may be multiple bodyparts in field of view 181 among which the server 116 may select one ormore for initial consideration. Alternatively, or in addition, server116 may select medical records which are related to the at least onebody part or region of the body of interest, and/or related to asuspected condition of the person in accordance with the user input.

In one example, server 116 may generate at least a first diagnosticprediction based upon the at least one condition of the at least onebody part in the visual data feed. The at least the first diagnosticprediction may further be based upon: at least the first medical record,at least the second medical record, and/or the user input. In oneexample, the server 116 may generate the first diagnostic prediction viaa machine learning-based pattern detection in accordance with the atleast one condition of the at least one body part. For instance, server116 may utilize one or more patterns or “signatures” stored in DB 118that may be used to identify different diagnostic predictions. Thepatterns/signatures may include various data points, or factors,including visual information from the visual feeds, information from theperson's medical records, and the medical records of relatives.

To illustrate, the visual information may include at least one conditionassociated with the at least one body part that is detected from thevisual data feed, such as a facial droop. The data points may alsoinclude medical records of the subject person and/or his or herrelative(s) which may indicate a history of stroke in the subjectindividual or in his/her family members, or a recent history of viralinfection. The server 116 may then match the at least one condition ofthe at least one body part in the visual data feed, at least the firstmedical record, and at least the second medical record to one or morepossible diagnostic predictions. For instance, the subject person mayhave Bell's palsy from viral infection or other causes, or may have hada stroke. If the subject person's medical records reveal recent viralinfection, or additional data (anonymized over many individuals) fromthe geographic area indicate that treatment for viral infection isprevalent in the area, the server 116 may be more likely to determine adiagnostic prediction of Bell's palsy. On the other hand, if familymedical records reveal that parents, siblings, or other relatives have ahistory of stroke, the server 116 may be more likely to determine adiagnostic prediction of stroke.

In addition, in one example, one or more user inputs may impact patternmatching/recognition by differentially weighting different factorsdepending upon the particular user input(s). For instance, a doctor'sinput regarding a suspected condition may bias the server 116 to be morelikely to find a certain pattern match associated with a diagnosticprediction. For instance, the subject person and/or a medicalprofessional may have greater reason to consider that the facialparalysis or facial droop may be stroke-related if there is awareness ofa personal or family history of stroke that may be inaccessible to theserver 116. Thus, a user input may bias the server 116 towards onediagnostic prediction or another.

The at least one diagnostic prediction may then be presented via theaugmented reality device (e.g., device 111). For instance, the at leastone diagnostic prediction may be projected in a transparent manner inthe visual field of user 191, e.g., overlaid over field of view 181,along with at least the first medical record and at least the secondmedical record.

It should be noted that the medical records may also be selected basedupon the user input. As such, there may be confirmation bias in terms ofthe input data selected. However, the present examples may be used as atool to assist a doctor or other medical professionals in obtainingrelevant medical records and achieving a diagnostic prediction. As such,the present disclosure is not intended to fully automate or replacecustomary interactions between medical professionals and patients intheir care.

In one example, server 116 may accept additional user inputs to selectadditional medical records, to focus on one or more different body partsor additional body parts that may be identified within a field of view,and/or medical records relating thereto, and so forth. In one example,server 116 may accept additional user inputs to rule out or excludesuspected conditions to dismiss one or more diagnostic predictions, andso forth. In such examples, server 116 may then obtain additionalmedical records, present the additional medical records via the ARdevice (e.g., device 111), provide one or more additional diagnosticpredictions, and so on. In other words, the server 116 may prioritizewhich medical record(s) to initially present in response to detecting abody part and/or movement related thereto. The server 116 can thenreceive one or more user inputs to call up additional medical records ifa user is not satisfied with the initially presented medical records orwould simply like to explore more medical records that may be related tothe at least one body part, if the user would like to obtain additionaldiagnostic predictions beyond that/those initially provided, and soforth.

In one example, server 116 may present via the AR device, e.g., one ofdevices 110 or 112, a recommendation to establish a visual communicationsession between the AR device and a device of a medical professional(e.g., device 113) based upon at least the first diagnostic prediction.In one example, the visual communication session may be established viathe AR device (e.g., device 110 or device 112) and the device of themedical professional (e.g., device 113), in response to an input fromthe person 190 or 192. Alternatively, the visual communication sessionmay be automatically established based upon at least the firstdiagnostic prediction, e.g., when at least the first diagnosticprediction includes a suspected urgent medical condition. These andother aspects of the present disclosure are discussed in greater detailbelow in connection with the examples of FIGS. 2 and 3.

Although only a single server 116 and a single DB 118 are illustrated,it should be noted that any number of servers 116 or databases 118 maybe deployed. In addition, server 116, DB 118, DB 106, server 104, and soforth may comprise public or private cloud computing resources, e.g.,one or more host devices/servers in one or more data centers to hostvirtual machines (VMs), containers, or the like comprising variousfunctions, services, and so on.

In one example, telecommunication network 102 may also include anapplication server 104 and a database 106. In one example, AS 104 mayperform the same or similar functions as server 116. Similarly, DB 106may store the same or similar information as DB 118, e.g., medicalrecords, indications of relations among individuals, informationregarding patterns for detecting body parts, particular motionsassociated with body parts, conditions of body parts, etc., machinelearning-based modules for making diagnostic predictions based upon bodypart conditions of subject persons, medical records of the persons,medical records of relatives, and/or user inputs, information forautomatically contacting medical providers or other caregivers on behalfof a person, and so forth, programs, logic, or instructions that may beexecuted by AS 104 or server 116 for presenting medical recordsassociated with a body part of a person via an augmented reality devicein accordance with the present disclosure, and so forth. For instance,telecommunication network 102 may provide a service for presentingmedical records associated with a body part of a person via an augmentedreality device to subscribers, e.g., in addition to television, phone,and/or other telecommunication services. In one example, AS 104, DB 106,server 116, and/or DB 118 may operate in a distributed and/orcoordinated manner to perform various steps, functions, and/oroperations described herein. In one example, application server 104 maycomprise network function virtualization infrastructure (NFVI), e.g.,one or more devices or servers that are available as host devices tohost virtual machines (VMs), containers, or the like comprising virtualnetwork functions (VNFs). In other words, at least a portion of thenetwork 102 may incorporate software-defined network (SDN) components.

It should be noted that the system 100 has been simplified. Thus, thesystem 100 may be implemented in a different form than that which isillustrated in FIG. 1, or may be expanded by including additionalendpoint devices, access networks, network elements, applicationservers, etc. without altering the scope of the present disclosure. Inaddition, system 100 may be altered to omit various elements, substituteelements for devices that perform the same or similar functions, combineelements that are illustrated as separate devices, and/or implementnetwork elements as functions that are spread across several devicesthat operate collectively as the respective network elements. Forexample, the system 100 may include other network elements (not shown)such as border elements, routers, switches, policy servers, securitydevices, gateways, a content distribution network (CDN) and the like.For example, portions of telecommunication network 102 and/or accessnetworks 120 and 122 may comprise a content distribution network (CDN)having ingest servers, edge servers, and the like.

Similarly, although only two access networks 120 and 122 are shown, inother examples, access networks 120 and/or 122 may each comprise aplurality of different access networks that may interface withtelecommunication network 102 independently or in a chained manner. Forexample, device 113 and server 116 may access telecommunication network102 via different access networks, devices 110-112 may accesstelecommunication network 102 via different access networks, and soforth. Thus, these and other modifications are all contemplated withinthe scope of the present disclosure.

FIG. 2 illustrates an example AR field of view in accordance withexamples of the present disclosure for presenting medical recordsassociated with a body part of a person via an augmented reality device.As illustrated in FIG. 2, the field of view 200 includes an image of aportion 210 of a person's body (e.g., including a forearm, hand, andwrist). As described above, the field of view 200 may be that of a uservia an AR device. For instance, the field of view 200 may be that of thesubject person examining the portion 210 of his or her own body, or maybe that of a medical provider examining the subject person. As alsodescribed above, the AR device and/or other components of network-basedprocessing system may identify at least one body part in a visual datafeed. For instance a camera of the AR device may capture the portion 210of the body in the field of view 200, and the AR device and/or othercomponents of a network-based processing system may identify thepresence of a forearm, wrist, hand, etc. via pattern recognition, e.g.,in accordance with one or more machine learning modules. In addition,the AR device and/or other components of a network-based processingsystem may obtain medical records of the subject person and one or morerelatives of the subject person (e.g., the medical records associatedwith at least one body part that is identified).

The medical records may then be presented via the AR device. Forinstance, the medical records may be presented in one or more layers, ortiles, e.g., as transparent visual overlays in/on the field of view 200.For example, medical records of the subject person may be presented in afirst tile, or layer 220, and medical records of the subject person'sfamily may be presented in a second tile, or layer 221. In the presentexample, the AR device and/or other components of a network-basedprocessing system may access medical records which indicate that thesubject person had a broken thumb in a prior year, e.g., the year 2014.Although not specifically related to the portion 210 of the body of thesubject person in the field of view 200, the medical records may alsoindicate that the subject person is currently taking the followingmedications: abc drug and xyz drug. This type of information may bedeemed important enough that it should be presented regardless of thespecific aspect of the body within the field of view 200.

Similarly, the AR device and/or other components of a network-basedprocessing system may access family medical records which indicate thatthe subject person's father was diagnosed with osteoporosis in 2009,which may be presented in summary form in layer 221. In one example,aspects of the information in the layers 220 and 221 may include linkswhich may provide additional information, for instance a user input viavoice command or other modalities (such as a gesture that may becaptured and recognized via a camera of the AR device) may select“x-rays” which may cause the actual x-ray images from the subjectperson's broken thumb to be displayed in the same layer or a new layer.

Additionally, the AR device and/or other components of a network-basedprocessing system may generate a diagnostic prediction on the basis ofthe visual data feed from the field of view 200, the personal and familymedical records, any additional user inputs, and so on. To illustrate,in the present example, the AR device and/or other components of anetwork-based processing system may identify a swollen pisiform boneprotrusion of the wrist from the visual data feed of the field of view200 (indicated by the region 230 in FIG. 2). Although the AR deviceand/or other components of a network-based processing system may bebiased to make a diagnostic prediction related to the subject person'sprior medical history or familial history (such as osteoporosis), in thepresent case the visual data indicates a swollen pisiform boneprotrusion, which appears to be unrelated to these prior personal andfamilial medical conditions. As such, the AR device and/or othercomponents of a network-based processing system may reach a diagnosticprediction of possible fracture unrelated to prior conditions. Inaddition, the diagnostic prediction may be presented in a third tile, orlayer 222 in/on the field of view 200 to inform the subject person ormedical professional utilizing the AR device.

It should be noted that the foregoing is just one example of the typesof human parts (e.g., limps, organs, etc.), medical histories, anddiagnostic predictions that may be determined and identified inaccordance with the present disclosure. For instance, in anotherexample, a field of view may include a face of a user, from which the ARdevice and/or other components of a network-based processing system mayidentify a face, may identify a condition of the face (e.g., facialdroop), may obtain medical records indicating a personal and/or familyhistory of stroke, may determine a diagnostic prediction of stroke basedupon the visual information and/or medical histories, and so on. Thus,these and other examples are all contemplated within the scope of thepresent disclosure.

FIG. 3 illustrates a flowchart of an example method 300 for presentingmedical records associated with a body part of a person via an augmentedreality device, in accordance with the present disclosure. In oneexample, the method 300 is performed by a component of the system 100 ofFIG. 1, such as by one of the server 116, application server 104, or anyof the devices 110-112, and/or any one or more components thereof (e.g.,a processor, or processors, performing operations stored in and loadedfrom a memory), or by one or more of the server 116, application server104, or any one of the devices 110-112 in conjunction with one or moreother devices, such as a different one or more of server 116,application server 104, or any one of the devices 110-112, and/or one ormore of DB 106, DB 118, device 114, and so forth. In one example, thesteps, functions, or operations of method 300 may be performed by acomputing device or system 400, and/or processor 402 as described inconnection with FIG. 4 below. For instance, the computing device orsystem 400 may represent any one or more components of a server 116,application server 104, and/or a device 110-112 in FIG. 1 that is/areconfigured to perform the steps, functions and/or operations of themethod 300. Similarly, in one example, the steps, functions, oroperations of method 300 may be performed by a processing systemcomprising one or more computing devices collectively configured toperform various steps, functions, and/or operations of the method 300.For instance, multiple instances of the computing device or processingsystem 400 may collectively function as a processing system. Forillustrative purposes, the method 300 is described in greater detailbelow in connection with an example performed by a processing system.The method 300 begins in step 305 and proceeds to step 310.

At optional step 310, the processing system may obtain a user inputidentifying at least one of a suspected condition of a person, apreference for a type of medical record, or a preference for medicalrecords associated with the at least one body part. The user input mayoriginate from the subject person or may originate from another user(e.g., a medical professional, a caregiver, or the like). For instance,the subject person, a medical professional, or other caregiver may beexamining the person via an AR device. In one example the AR devicecomprises the processing system. Alternatively, or in addition, theprocessing system may comprise a network-based processing system incommunication with the AR device.

At step 320, the processing system identifies at least one body part ofa person in a visual data feed of the AR device. The AR device may beused by the subject person or by a medical professional or othercaregivers while examining the person. In one example, the visual datafeed may be captured by an outward-facing camera of the AR device thatis directed at the at least one body part. In one example, theidentifying of the at least one body part comprises detecting a motionassociated with the at least one body part. For instance, the motion maycomprise a cough, a heartbeat pattern, a vascular/blood flow pattern, amovement pattern of a mouth during speech, a tremor in the at least onebody part (e.g., one or more body parts), and so forth. In one example,the identifying the at least one body part is via a machinelearning-based pattern detection in accordance with information from thevisual data feed. In one example, the identifying the at least one bodypart comprises determining at least one condition associated with the atleast one body part based upon the visual data feed. For instance, theprocessing system may detect a mole on the skin, a mole pattern, asunburn or sunburn pattern, a skin infection, a lesion, a swollen joint,swollen skin, etc. Similarly, the processing system may detect a pulsearrhythmia, a facial paralysis and/or facial droop, and so forth. In oneexample, the identifying the at least one condition of the at least onebody part is via a machine learning-based pattern detection inaccordance with information from the visual data feed.

At step 330, the processing system obtains at least a first medicalrecord of the person that is associated with the at least one body part.In one example, step 330 may include selecting at least the firstmedical record from among a plurality of medical records of the personbased upon the at least one body part that is identified at step 320and/or the user input that may be obtained at optional step 310. In oneexample, at least the first medical record may be selected further basedupon a movement related to the at least one body part that may bedetected at step 320 and/or at least one condition associated with thebody part that may be detected at step 320. In other words, theprocessing system may prioritize which medical record(s) of the subjectperson to initially obtain (e.g., for presentation at step 360) inresponse to detecting the body part and/or a movement or conditionrelated thereto.

At step 340, the processing system obtains at least a second medicalrecord of at least one relative of the person that is associated withthe at least one body part. In one example, step 340 may includeselecting at least the second medical record from among a plurality ofmedical records of the at least one relative based upon the at least onebody part that is identified at step 320 and the user input that may beobtained at optional step 310. In one example, at least the secondmedical record may be selected further based upon a movement related tothe at least one body part that may be detected at step 320 and/or atleast one condition associated with the body part that may be detectedat step 320. In other words, the processing system may prioritize whichmedical record(s) of the at least one relative to initially obtain(e.g., for presentation at step 360) in response to detecting the bodypart and/or a movement or condition related thereto. It should be notedthat any presentation of medical records of any individuals must bepreviously authorized by those individuals, e.g., parents of a child maypreauthorize the release of their medical records in assisting thetreatment of their child. However, in order to ensure that theindividuals' confidential medical information is protected, positive andspecific preauthorization must be received prior to the usage of suchmedical information from such individuals even though the patient isrelated to such individuals.

At optional step 350, the processing system may generate at least afirst diagnostic prediction based upon the at least one condition of theat least one body part in the visual data feed. The at least the firstdiagnostic prediction may further be based upon one or both of at leastthe first medical record or at least the second medical record. In oneexample, the first diagnostic prediction is generated via a machinelearning-based pattern detection in accordance with the at least onecondition of the at least one body part in the visual data feed. Forinstance, the processing system may utilize one or more patterns or“signatures” stored in a database accessible to the processing systemthat may be used to identify different diagnostic predictions. Thepatterns/signatures may include various data points, or factors,relating to visual information from visual feeds, information from asubject person's medical records, and the medical records of relatives.In one example, at least the first diagnostic prediction may further bebased on user (e.g., one or more medical professionals) input that maybe obtained at optional step 310. For instance, if the information fromthe visual data feed is consistent with a suspected medical condition,the processing system may be more likely to generate at least the firstdiagnostic prediction that is consistent with the suspected medicalcondition. In other words, at least the first diagnostic prediction maycomprise the suspected medical condition. However, where the visual datafeed, at least the first medical record, and/or at least the secondmedical record are inconsistent with or would tend to indicate that thesuspected medical condition is not present, the processing system maygenerate a different diagnostic prediction that is consistent with theavailable information of the visual data feed, at least the firstmedical record, and/or at least the second medical record.

To illustrate, the least one condition detected from the visual datafeed may comprise a facial droop. The data points may also includemedical records of the subject person and/or his or her relative(s) (whopreauthorized such use) which may indicate a history of stroke in thesubject individual or in his/her family members, or a recent history ofviral infection. The processing system may then match the at least onecondition of the at least one body part in the visual data feed, atleast the first medical record, and at least the second medical record,to one or more possible diagnostic predictions. For instance, thesubject person may have Bell's palsy from viral infection or othercauses, or may have had a stroke. If the subject person's medicalrecords reveal recent viral infection, or additional data (e.g.,anonymized over many unknown individuals and/or preauthorized from suchindividuals) from a relevant geographic area (e.g., a city, a county, astate, etc.) indicate that treatment for viral infection is prevalent inthe area, the processing system may be more likely to determine adiagnostic prediction of Bell's palsy. On the other hand, if familymedical records reveal that parents, siblings, or other relatives have ahistory of stroke, the processing system may be more likely to determinea diagnostic prediction of stroke. In addition, in one example, one ormore user inputs may impact pattern matching/recognition bydifferentially weighting different factors. For instance, a suspectedcondition may bias the processing system to be more likely to find acertain pattern match associated with a diagnostic prediction.

At step 360, the processing system presents, via the AR device, at leastthe first medical record and at least the second medical record. In oneexample, step 360 further comprises presenting at least the firstdiagnostic prediction of optional step 350 via the AR device. In oneexample, step 360 may comprise projecting a transparent overlay of atleast the first medical record and at least the second medical recordvia the AR device. For instance, an example of transparent overlay viaan AR device is illustrated in FIG. 2.

At optional step 370, the processing system may present, via the ARdevice, a recommendation to establish a visual communication sessionbetween the AR device and a device of a medical professional based uponat least the first diagnostic prediction of optional step 350 (e.g., inan example where the AR device is used by the person). For example, atleast the first diagnostic prediction may relate to a suspected medicalcondition for which examination by a medical professional is deemedwarranted. For instance, a database available to the processing systemmay store indications for which given diagnostic predictions shouldresult in a recommendation for communication with a medicalprofessional.

At optional step 380, the processing system may establish, via the ARdevice, the visual communication session with the device of the medicalprofessional. In one example, the visual communication session may beestablished in response to an input from the person (e.g., having beenpresented with the recommendation at optional step 370). In anotherexample, the visual communication session may be established based uponat least the first diagnostic prediction (e.g., automatically, withoutspecific user input). For instance, the establishment of the visualcommunication session may be automatic when at least the firstdiagnostic prediction includes a suspected urgent medical condition. Forinstance, a database available to the processing system may storeindications for which given diagnostic predictions should result inautomatic communication with a medical professional.

Following step 350 or any one or more of optional steps 360-380 themethod 300 proceeds to step 395 where the method ends.

It should be noted that the method 300 may be expanded to includeadditional steps, or may be modified to replace steps with differentsteps, to combine steps, to omit steps, to perform steps in a differentorder, and so forth. For instance, in one example the processing systemmay repeat one or more steps of the method 300, such as steps 320-360,steps 310-380, etc. For example, a user may direct a camera of the ARdevice toward one or more additional or different body parts, which mayresult in obtaining different medical records, different diagnosticpredictions, and so forth. In another example, the method 300 may beexpanded to include receive one or more additional user inputs to callup additional medical records, to dismiss one or more of the at leastone diagnostic prediction, to provide one or more additional suspectedmedical conditions, and so forth. For instance, the AR device and/or theprocessing system may accept and respond to user inputs if a user is notsatisfied with the initially presented medical records, if the userwould like to explore more medical records that may be related to the atleast one body part, if the user would like to examine the presentedmedical records in more detail, and so on. Thus, these and othermodifications are all contemplated within the scope of the presentdisclosure.

In addition, although not expressly specified above, one or more stepsof the method 300 may include a storing, displaying and/or outputtingstep as required for a particular application. In other words, any data,records, fields, and/or intermediate results discussed in the method canbe stored, displayed and/or outputted to another device as required fora particular application. Furthermore, operations, steps, or blocks inFIG. 3 that recite a determining operation or involve a decision do notnecessarily require that both branches of the determining operation bepracticed. In other words, one of the branches of the determiningoperation can be deemed as an optional step. However, the use of theterm “optional step” is intended to only reflect different variations ofa particular illustrative embodiment and is not intended to indicatethat steps not labelled as optional steps to be deemed to be essentialsteps. Furthermore, operations, steps or blocks of the above describedmethod(s) can be combined, separated, and/or performed in a differentorder from that described above, without departing from the exampleembodiments of the present disclosure.

FIG. 4 depicts a high-level block diagram of a computing device orprocessing system specifically programmed to perform the functionsdescribed herein. For example, any one or more components or devicesillustrated in FIG. 1 or described in connection with the method 300 maybe implemented as the processing system 400. As depicted in FIG. 4, theprocessing system 400 comprises one or more hardware processor elements402 (e.g., a microprocessor, a central processing unit (CPU) and thelike), a memory 404, (e.g., random access memory (RAM), read only memory(ROM), a disk drive, an optical drive, a magnetic drive, and/or aUniversal Serial Bus (USB) drive), a module 405 for presenting medicalrecords associated with a body part of a person via an augmented realitydevice, and various input/output devices 406, e.g., a camera, a videocamera, storage devices, including but not limited to, a tape drive, afloppy drive, a hard disk drive or a compact disk drive, a receiver, atransmitter, a speaker, a display, a speech synthesizer, an output port,and a user input device (such as a keyboard, a keypad, a mouse, and thelike).

Although only one processor element is shown, it should be noted thatthe computing device may employ a plurality of processor elements.Furthermore, although only one computing device is shown in the Figure,if the method(s) as discussed above is implemented in a distributed orparallel manner for a particular illustrative example, i.e., the stepsof the above method(s) or the entire method(s) are implemented acrossmultiple or parallel computing devices, e.g., a processing system, thenthe computing device of this Figure is intended to represent each ofthose multiple computers. Furthermore, one or more hardware processorscan be utilized in supporting a virtualized or shared computingenvironment. The virtualized computing environment may support one ormore virtual machines representing computers, servers, or othercomputing devices. In such virtualized virtual machines, hardwarecomponents such as hardware processors and computer-readable storagedevices may be virtualized or logically represented. The hardwareprocessor 402 can also be configured or programmed to cause otherdevices to perform one or more operations as discussed above. In otherwords, the hardware processor 402 may serve the function of a centralcontroller directing other devices to perform the one or more operationsas discussed above.

It should be noted that the present disclosure can be implemented insoftware and/or in a combination of software and hardware, e.g., usingapplication specific integrated circuits (ASIC), a programmable logicarray (PLA), including a field-programmable gate array (FPGA), or astate machine deployed on a hardware device, a computing device, or anyother hardware equivalents, e.g., computer readable instructionspertaining to the method(s) discussed above can be used to configure ahardware processor to perform the steps, functions and/or operations ofthe above disclosed method(s). In one example, instructions and data forthe present module or process 405 for presenting medical recordsassociated with a body part of a person via an augmented reality device(e.g., a software program comprising computer-executable instructions)can be loaded into memory 404 and executed by hardware processor element402 to implement the steps, functions or operations as discussed abovein connection with the example method 300. Furthermore, when a hardwareprocessor executes instructions to perform “operations,” this couldinclude the hardware processor performing the operations directly and/orfacilitating, directing, or cooperating with another hardware device orcomponent (e.g., a co-processor and the like) to perform the operations.

The processor executing the computer readable or software instructionsrelating to the above described method(s) can be perceived as aprogrammed processor or a specialized processor. As such, the presentmodule 405 for presenting medical records associated with a body part ofa person via an augmented reality device (including associated datastructures) of the present disclosure can be stored on a tangible orphysical (broadly non-transitory) computer-readable storage device ormedium, e.g., volatile memory, non-volatile memory, ROM memory, RAMmemory, magnetic or optical drive, device or diskette and the like.Furthermore, a “tangible” computer-readable storage device or mediumcomprises a physical device, a hardware device, or a device that isdiscernible by the touch. More specifically, the computer-readablestorage device may comprise any physical devices that provide theability to store information such as data and/or instructions to beaccessed by a processor or a computing device such as a computer or anapplication server.

While various embodiments have been described above, it should beunderstood that they have been presented by way of example only, and notlimitation. Thus, the breadth and scope of a preferred embodiment shouldnot be limited by any of the above-described example embodiments, butshould be defined only in accordance with the following claims and theirequivalents.

1.-20. (canceled)
 21. A method comprising: obtaining, by a processingsystem including at least one processor, a user input identifying atleast one of: a suspected condition of a person; a preference for a typeof medical record; or a preference for medical records associated withat least one body part; identifying, by the processing system, the atleast one body part of the person in a visual data feed of an augmentedreality device, wherein the identifying the at least one body partcomprises determining at least one condition associated with the atleast one body part based upon the visual data feed; obtaining based onthe identifying, by the processing system, at least a first medicalrecord of the person that is associated with the at least one body part;obtaining, by the processing system, at least a second medical record ofat least one relative of the person, wherein the obtaining the at leastthe second medical record comprises selecting the at least the secondmedical record from among a plurality of medical records of the at leastone relative based upon the user input; generating, by the processingsystem, at least a first diagnostic prediction via a first machinelearning-based pattern detection based upon the at least one conditionof the at least one body part in the visual data feed, the at least thefirst medical record, and the at least the second medical record; andpresenting, by the processing system via the augmented reality device,the at least the first medical record, the at least the second medicalrecord, and the at least the first diagnostic prediction, wherein thepresenting comprises projecting a transparent overlay of the at leastthe first medical record, the at least the second medical record, andthe at least the first diagnostic prediction via the augmented realitydevice.
 22. The method of claim 21, where the identifying the at leastone body part comprises detecting a motion associated with the at leastone body part.
 23. The method of claim 21, wherein the obtaining the atleast the first medical record comprises selecting the at least thefirst medical record from among a plurality of medical records of theperson based upon the at least one body part that is identified and theuser input.
 24. The method of claim 21, wherein the identifying the atleast one body part is via at least a second machine learning-basedpattern detection in accordance with information from the visual datafeed.
 25. The method of claim 24, wherein the identifying the at leastone condition of the at least one body part is via the at least thesecond machine learning-based pattern detection in accordance with theinformation from the visual data feed.
 26. The method of claim 21,further comprising: presenting, via the augmented reality device, arecommendation to establish a visual communication session between theaugmented reality device and a device of a medical professional basedupon the at least the first diagnostic prediction, wherein the augmentedreality device is used by the person.
 27. The method of claim 26,further comprising: establishing, via the augmented reality device, thevisual communication session with the device of the medicalprofessional, in response to an input from the person.
 28. The method ofclaim 21, further comprising: establishing, via the augmented realitydevice, a visual communication session with a device of a medicalprofessional, based upon the at least the first diagnostic prediction.29. The method of claim 21, wherein the augmented reality devicecomprises the processing system.
 30. The method of claim 21, wherein theprocessing system is a network-based processing system in communicationwith the augmented reality device.
 31. A non-transitorycomputer-readable medium storing instructions which, when executed by aprocessing system including at least one processor, cause the processingsystem to perform operations, the operations comprising: obtaining auser input identifying at least one of: a suspected condition of aperson; a preference for a type of medical record; or a preference formedical records associated with at least one body part; identifying theat least one body part of the person in a visual data feed of anaugmented reality device, wherein the identifying the at least one bodypart comprises determining at least one condition associated with the atleast one body part based upon the visual data feed; obtaining based onthe identifying at least a first medical record of the person that isassociated with the at least one body part; obtaining at least a secondmedical record of at least one relative of the person, wherein theobtaining the at least the second medical record comprises selecting theat least the second medical record from among a plurality of medicalrecords of the at least one relative based upon the user input;generating at least a first diagnostic prediction via a first machinelearning-based pattern detection based upon the at least one conditionof the at least one body part in the visual data feed, the at least thefirst medical record, and the at least the second medical record; andpresenting, via the augmented reality device, the at least the firstmedical record, the at least the second medical record, and the at leastthe first diagnostic prediction, wherein the presenting comprisesprojecting a transparent overlay of the at least the first medicalrecord, the at least the second medical record, and the at least thefirst diagnostic prediction via the augmented reality device.
 32. Anapparatus comprising: a processing system including at least oneprocessor; and a computer-readable medium storing instructions which,when executed by the processing system, cause the processing system toperform operations, the operations comprising: obtaining a user inputidentifying at least one of: a suspected condition of a person; apreference for a type of medical record; or a preference for medicalrecords associated with at least one body part; identifying the at leastone body part of the person in a visual data feed of an augmentedreality device, wherein the identifying the at least one body partcomprises determining at least one condition associated with the atleast one body part based upon the visual data feed; obtaining based onthe identifying at least a first medical record of the person that isassociated with the at least one body part; obtaining at least a secondmedical record of at least one relative of the person, wherein theobtaining the at least the second medical record comprises selecting theat least the second medical record from among a plurality of medicalrecords of the at least one relative based upon the user input;generating at least a first diagnostic prediction via a first machinelearning-based pattern detection based upon the at least one conditionof the at least one body part in the visual data feed, the at least thefirst medical record, and the at least the second medical record; andpresenting, via the augmented reality device, the at least the firstmedical record, the at least the second medical record, and the at leastthe first diagnostic prediction, wherein the presenting comprisesprojecting a transparent overlay of the at least the first medicalrecord, the at least the second medical record, and the at least thefirst diagnostic prediction via the augmented reality device.
 33. Theapparatus of claim 32, where the identifying the at least one body partcomprises detecting a motion associated with the at least one body part.34. The apparatus of claim 32, wherein the obtaining the at least thefirst medical record comprises selecting the at least the first medicalrecord from among a plurality of medical records of the person basedupon the at least one body part that is identified and the user input.35. The apparatus of claim 32, wherein the identifying the at least onebody part is via at least a second machine learning-based patterndetection in accordance with information from the visual data feed. 36.The apparatus of claim 32, the operations further comprising:presenting, via the augmented reality device, a recommendation toestablish a visual communication session between the augmented realitydevice and a device of a medical professional based upon the at leastthe first diagnostic prediction, wherein the augmented reality device isused by the person.
 37. The apparatus of claim 35, wherein theidentifying the at least one condition of the at least one body part isvia the at least the second machine learning-based pattern detection inaccordance with the information from the visual data feed.
 38. Theapparatus of claim 36, the operations further comprising: establishing,via the augmented reality device, the visual communication session withthe device of the medical professional, in response to an input from theperson.
 39. The apparatus of claim 32, the operations furthercomprising: establishing, via the augmented reality device, a visualcommunication session with a device of a medical professional, basedupon the at least the first diagnostic prediction.
 40. The apparatus ofclaim 32, wherein the augmented reality device comprises the processingsystem.